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Preliminary Impacts of North Carolina’s Rural Innovative Schools Project 

Background and Context: Funded by an Investing in Innovation (i3) Validation grant, the Rural 
Innovative Schools (RIS) Project is the first widespread effort to scale up the early college model by 
implementing it in comprehensive high schools. This paper will present preliminary findings from 
the evaluation of this project. 

Purpose and Goals of the Study: This study is designed to examine the impact of the early college 
strategies as implemented in the Rural Innovative Schools Project. This paper will report on the 
following research question: 

To what extent does participation in the RIS Project result in improved outcomes for students 
including: increased student enrollment and success in college-credit bearing courses (dual 
credit and AP), improved graduation rates, improved student attendance, decreased dropout 
rates, and improved student enrollment and success in college preparatory courses? 

Setting: In this project, the early college model is being implemented in existing comprehensive 
high schools in 10 rural, low-income school districts across the state of North Carolina. 

Sample: The sample included in this study is a total of 18 treatment schools matched to 18 
comparison schools in North Carolina. Five treatment schools began their participation in the project 
in 2012-2013 and 13 schools began participating in the 2013-2014 school year. Table 1 presents an 
overview of the baseline characteristics of participating schools compared with the state averages. 

TABLE 1 HERE 

Intervention: The Rural Innovative Schools Project is designed to increase the number of students 
who graduate from high school and are prepared for enrollment and success in postsecondary 
education. The Key Components of the Rural Innovative Schools Project include a set of services 
that are intended to support implementation of a whole-school reform model emphasizing the 
creation of a college -preparatory school environment through six Design Principles. The first Key 
Component is Integrated Systems of Support that include four specific activities: 1 .) a series of 
professional development activities centered around implementation of the six Design Principles; 2.) 
on-site leadership coaching for administrative teams on the Design Principles; 3.) on-site 
instructional coaching on the Design Principles, emphasizing the Common Instructional Framework; 
and 4.) staff support for the schools. The second Key Component is Support for College Credit 
Courses and includes 5.) funding for college credit courses for students; 6.) funding for a college 
liaison; and 7.) assistance in developing partnerships with postsecondary institutions. The final Key 
Component is Activities to Influence Context, which includes: 8.) community development work; 9.) 
professional development with the district; and 10.) dissemination activities. As a result of these 
services, each school is expected to implement six Design Principles that represent characteristics of 
an effective high school. These Design Principles, as articulated by North Carolina New Schools, are 
as follows: 1.) Ensuring that students are ready for college; 2.) Instilling powerful teaching and 
learning in schools; 3.) Providing high student/staff personalization; 4. ) Redefining professionalism; 
5.) Creating leadership that develops a collective vision; and 6.) Implementing a purposeful design in 
which school structures support all of the above principles. A primary emphasis of the program will 
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be increasing the number of students who participate in college credit-bearing courses while in high 
school. Figure 1 presents the logic model for the Rural Innovative Schools Project. 

FIGURE 1 HERE 

Research Design: The impact study uses a quasi-experimental design to assess the impact of the 
RIS Project on a core set of student outcomes. The treatment schools were matched to schools that 
were equivalent on a core set of baseline characteristics. To determine baseline equivalence, we used 
the same analytic approach as for the outcomes (see below), except that no covariates were included. 
As table 2 shows, the groups were equivalent on all baseline characteristics at levels less than .25 
standard deviations. Pursuant to What Works Clearinghouse standards (Institution of Education 
Sciences, March, 2014), for each outcome analyses, we include as covariates each baseline measure 
where the difference is between 0.05 and 0.25 standard deviations. 

TABLE 2 HERE 

Data Collection: The study is using data collected from the schools by the North Carolina 
Department of Public Instruction and housed at the North Carolina Education Research Data Center 
at Duke University. The specific outcomes examined in this study include: 

Percent of students who have enrolled in at least 1 College-Credit Bearing- Course by the end of 
11th grade. A primary goal of the intervention is to increase the number of students who have 
access to college-credit-bearing courses. This measure is therefore designed to look at the percentage 
of the student body that is given access to these courses. For purposes of this study, we are looking 
at any course that has the potential to bear college credit, including Advanced Placement, 
International Baccalaureate and dual enrollment courses. A student will be coded as taking a 
college-credit-bearing course if they have enrolled in at least one AP, IB, or dual enrollment course 
by the end of 1 1 th grade. 

Average number of college-credit bearing courses students have taken and passed by the end of 12 th 
grade. The previous measure spoke to access; this measure tries to get at the depth of the students’ 
experiences with college credit through the number of courses successfully completed. NC New 
Schools has a goal of having at least 50% of students successfully completing at least 21 college 
credits. Student can receive college credit in two ways. The first way is to receive a grade of D or 
higher in a dual enrollment course, which is the level for which a student can receive college credit. 
The second way is to receive a level 3 or higher on the AP exam (which is the level accepted by 
many colleges). We do not include IB courses, because we do not have access to IB exam scores; 
however, given the relatively small number of North Carolina students who have taken IB exams 
(approximately 5,000 in 2009, the vast majority of which were in urban areas that are not a part of 
this study), we do not expect this to be problematic. 

Cohort graduation rate. This outcome is a school-level outcome. Schools report their graduating 
students to NCPDI. As noted above, NCDPI calculates a cohort graduation rate for each school by 
identifying the students who were enrolled in 9 th grade in the school four years earlier, removing any 
students who transferred schools, and detennining whether each student is represented in the 
graduation file. Students who dropped out or are missing and unaccounted for are included as non- 
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graduates. We will use the school-level data because NCDPI is already using the approach that we 
would use and they produce the official cohort graduation rates for the school. 

Attendance. Student attendance has been positively associated with progress in school (Lee & 
Burkham, 2003); changes in student attendance are therefore seen as a reliable indicator of students’ 
likelihood of remaining in school. The evaluation will examine the number of days that a student is 
absent from school. 

Dropout rate. This measure examines the dropout rate for each school. Students in the dropout file 
are students who either completed a fonn indicating that they are dropping out of school or had the 
school indicate that they dropped out. Students who are not listed in the dropout file are considered 
not to have dropped out. 

College preparatory course-taking. This measure looks at the proportion of students taking a core 
set of college preparatory courses at the 9 th grade level. The courses to be examined include those 
that would ensure that a student is on-track for entrance into the University of North Carolina 
system. In 9 th grade, these courses include English I and at least one college preparatory 
mathematics course (Algebra I, Geometry, Algebra II, Integrated Math I, Common Core Math I 
or higher). Research has shown that it is extremely challenging for students who are off-track for 
college in 9 th grade to catch up (Finkelstein & Fong, 2008). Examining the percentage of 
students taking these courses is a measure of the extent to which the school provides access to 
courses needed for college to a wide range of students. Students who dropped out will be 
assumed to have not taken college prep courses after dropping out. 

College preparatory course-taking and success. This measure is very closely related to the 
previous measure and is the percentage of students taking and succeeding in English I and at least 
one college preparatory math course in the 9 th grade. Successful completion will be defined as 
passing the course with a grade of C or higher. While the first measure speaks to access, this 
second measure of successful course completion captures both access and success in school and 
does not penalize schools that are expanding access to more students. Students who dropped out 
will be assumed to have not taken college prep courses after dropping out. 

Analysis: We use hierarchical linear modeling (HLM) (Raudenbush & Bryk, 2002) as the general 
analytic framework within which we will examine impacts of interest to account for the nested 
structure of the data. In most cases, where student-level data is available, students will be nested 
within schools. Schools are also nested within districts, however, because the sample is comprised of 
rural districts there is typically only one high school per district. Due to such confounding and 
because we have no substantive questions related to district effects, we do not model district-level 
nesting. For binary outcomes, we use linear probability modeling (LPM) because these are 
appropriate and return similar results to alternative non-linear specifications that involve logistic or 
probit regressions as long as most of the observations are not in close vicinity of 0 or 1 . Compared to 
nonlinear specifications, results from linear probability models are more easily interpreted because 
the estimated coefficients yield marginal effects (Angrist & Pischke, 2008). 

Results: We are in the process of finalizing the analyses for the 2012-2013 results for Cohort 1. We 
have constructed the datasets for the 2013-2014 school year and will have completed analyses prior 
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to the spring presentation. Thus, our conference presentation will include impact estimates for the 
first two years of implementation for the first cohort of schools and for the first year of 
implementation for the second two cohorts of schools. We will present results for all of the outcomes 
noted above. 

Conclusions: The results from this study, particularly when coupled with evaluations of the other i3 
projects, will show the extent to which implementation of early college strategies in comprehensive 
high schools can have the same impact as the stand-alone early college model. 
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Appendix B: Tables and Figures 


Table 1: Background Characteristics of Participating Schools 


Treatment 

Cohort 

Student 

Enrollment 

% 

Students 

in 

Poverty 

% Under- 
represented 
Minority 
Students 

On-Time 
Grad Rate 
Avg (5 
year 
average) 

Percent 
Enrollments 
in AP/IB/ 
College 
Credit 
Courses 

Attendanc 
e Rate 

Avg. Pass 
Rate 
Core 
Subjects 

Cohort 1 (11-12) 

615 

59.3% 

23.8% 

74.0% 

2.1% 

94.5% 

80.0% 

State Average 
(11-12) 

943 

48.1% 

41.4% 

79.1% 

5.2% 

94.4% 

79.8% 

Cohort 2 (12-13) 

708 

54.5% 

29.5% 

78.1% 

2.9% 

94.3% 

40.2% 

Cohort 3 (12-13) 

509 

68.4% 

61.7% 

74.6% 

2.4% 

92.9% 

28.3% 

State Average 
(12-13) 

975 

48.4% 

41.1% 

81.2% 

5.2% 

94.2% 

39.4% 


Note. State averages calculated using only traditional public high schools that serve grades 9 -12 (Source: NC Report 
Card Data). All non-traditional high schools were removed from the analytic file to provide a more accurate comparison 
of cohort descriptives to state averages. 
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Figure 1: Rural Innovative Schools Logic Model 



Contextual Factors influencing Implementation 

Career and College Promise policies. Common Core, Alignment to existing district initiatives, Leadership's openness to 

change. Staff and community buy-in 
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Table 2: Baseline equivalence for treatment and comparison groups 


Baseline Measure 

Treatment 

Comparison 

Effect Size 

% course enrollments in 
AP/IB/Dual credit 

2.66% 

2.52% 

-0.13* 

Graduation Rate 

80.54% 

81.72% 

0.16 

Attendance Rate 

93.83% 

93.59% 

-0.17* 

Dropouts 

3.24% 

3.35% 

.07* 

% enrolled and 
succeeding in college 
preparatory courses 

82.07% 

81.80% 

-.01 

% poverty 3 

58.79% 

57.68% 

-.10* 

% underrepresented 
minority 3 

36.14% 

37.85% 

.07* 


a All outcomes with the exception of graduation rates are calculated by aggregating student-level data. Because different outcomes have different levels 
of missing data associated with them, the percentages of poverty and underrepresented minority students vary slightly for the samples used in the 
different outcomes. The percentages presented here are associated with the first outcome— % course enrollments in advanced courses. 
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